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ABSTRACT 


Five  government  funded  parallel  research  and  development 
projects  plus  four  single  projects  are  examined  to  determine 
the  manner  in  which  engineers  and  scientists  allocate  their 
time,  and  the  effect  of  this  allocation  on  the  outcome  of 
the  projects.   The  use  of  matched  pairs  of  projects  allows 
the  relative  evaluation  of  outcomes  by  technical  monitors 
in  the  customer  agencies. 

In  the  research  projects  the  higher  rated  teams  spend 
more  time  than  the  lower  rated  teams  in  literature  search 
and  staff  consulting.   The  percent  of  total  time  spent  in 
three  categories  of  information  gathering  (outside  consul- 
tation, staff  consultation,  and  literature  search)  varies 
significantly  over  the  life  of  a  project.   Higher  rated 
teams  are  relatively  stable  in  all  phases  of  information 
gathering  while  lower  rated  teams  initially  spend  far  more 
time  gathering  information  than  they  do  in  the  later  stages 
and  fluctuate  more  throughout  the  project.   Subsystems 
characterized  by  greater  uncertainty  receive  a  higher  per- 
cent of  information  gathering  time  than  subsystems  where  un- 
certainty is  lower. 


INTRODUCTION 

Research  and  development  is  a  problem  solving  process  depending 
on  information  inputs  from  the  environment  as  well  as  on  the  stored 
knowledge  and  the  ability  of  the  personnel  involved.   Improvement 
in  the  flow  of  scientific  information  is  an  important  goal  of  R<?,D 
management,  but  is  dependent  upon  a  thorough  understanding  of  the 
process  itself.   The  research  reported  here  is  concerned  with  the 
manner  in  which  individual  engineers  and  scientists  allocate  their 
time  among  three  general  classes  of  information  channel  (literature; 
personnel  outside  the  laboratory;  and  personnel  within  the  lab)  and 
examines  possible  effects  this  allocation  mav  have  on  the  conduct 
and  performance  of  the  R<!,D  project  on  which  they  are  engaged. 

In  order  to  obtain  a  performance  evaluation  for  each  informa- 
tion channel,  instances  were  sought  in  which  the  same  problem  is 

I 
attempted  by  two  or  more  research  groups.    In  this  way,  a  compari- 
son can  be  made  between  the  information  sources  leading  to  specific 
solutions.   The  fairly  common  practice  of  the  Department  of  Defense 
and  the  NASA  to  award  parallel  study  contracts  during  the  early  phases 
of  R&D  provides  an  opportunity  for  control  over  the  effect  of  varia- 


The  idea  of  studying  parallel  R&D  projects  as  a  quas i -exper i  • 
mental  situation  was  suggested  by  Donald  Marquis,  who  also  helped 
in  developing  some  of  the  research  procedures. 
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tions  in  the  problem  substance.  The  data  presented  are  from  five 
parallel  contracts  involving  ten  RiD  groups  and  four  single  pro- 
jects. 

The  major  hypotheses  of  this  study  concern  the  variation  in 
time  allocated  among  the  three  information  channels  over  the  course 
of  an  R&D  project.   Specifically,  it  is  hypothesized  that: 

1.  Literature  will  be  used  more  heavily  at  the  initiation 
of  a  project  and  will  be  replaced  by  personnel  outside 
the  laboratory  (principally  vendors)  as  the  principal 
information  source  as  the  project  progresses, 

2.  Sources  within  the  lab  will  be  employed  intermittently 
over  the  course  of  a  project. 

3.  Teams  conducting  an  extensive  information  search  at  the 
beginning  of  a  project  perform  better, 

4.  The  extent  to  which  the  lab's  technical  staff  is  used  as 
an  information  source  will  be  d  i  rect ly  related  to  perfor- 
mance. 

5.  The  extent  to  which  the  team  relies  upon  individuals  out- 
side of  the  laboratory  organization  as  information  sources 
will  be  i  nverse  I  y  related  to  performance. 

6.  Subsystems  with  a  high  degree  of  uncertainty  (uncertainty 

is  operationally  defined)  receive  greater  attention  in  terms 
of  time  spent  on  information  gathering  than  those  having  a 
lower  degree  of  uncertainty. 


METHOD 

Once  a  parallel  project  has  been  located,  its  work  statement 
is  obtained,  and  analyzed  and  factored  into  a  reasonable  number  of 
subproblem  areas  (generally  subsystems).   The  breakdown  is  then 
checked  with  the  technical  person  who  prepared  the  work  statement, 
and  data  collection  forms  based  upon  it  are  designed.  After  all 
data  have  been  collected  from  the  contractors,  the  technical  moni- 
tor is  revisited  and  asked  to  provide  a  confidential  evaluation  of 
each  lab's  performance.   Data  are  gathered  by  four  means:   (I)  time 
allocation  forms,  indicating  the  amount  of  time  each  engineer  spends 
on  the  job  in  several  activity  categories;  (2)  before  and  after  inter- 
views with  the  individual  engineers;  (3)  periodic  tape  recorded  pro- 
gress reports  by  the  project  manager;  and  (4)  solution  development 
records. 

Time  allocation  forms  supported  by  data  from  Solution  Develop- 
ment Records  are  the  principal  source  of  the  data  here  under  con- 
sideration.  On  these  forms,  respondents  are  asked  simply  to  record 
the  amount  of  time  spent  each  day  in  each  of  four  activity  categories: 
time  spent  in  literature  search;  time  spent  in  consultation  with  tech- 
nical experts  within  the  laboratory,  but  not  assigned  to  the  project; 
time  spent  in  consulting  with  experts  outside  the  laboratory  organi- 
zation; and  time  spent  in  analytic  design  or  active  problem  solving. 
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The  Solution  Development  Record  provides  a  means  of  following  the 
progress  of  an  R&D  project  through  its  subsystem  design  changes, 
and  has  been  described  in  detail  elsev/here  (Alien,  1966). 

Description  of  the  Projects 

The  five  parallel  projects  under  consideration  involved  the 
following  general  problems: 

1.  The  design  of  the  reflector  portion  of  a  rather  large 
and  highly  complex  antenna  system  for  tracking  and  com- 
munication with  space  vehicles  at  very  great  distances. 

2.  The  design  of  a  vehicle  and  associated  instrumentation 

to  roam  the  lunar  surface  and  gather  descriptive  scienti- 
fic data. 

3.  The  study  of  possible  mission  profiles  for  manned  flights 
to  another  p  lanet . 

4.  The  development  of  a  detailed  mathematical  cost  model 
yielding  complete  R&D  and  operational  program  cost  esti- 
mates of  space  launch  vehicle  systems. 

5.  An  investigation  of  passive  methods  for  transfer  of  modula- 
tion between  two  coherent  light  beams. 

In  addition  data  from  four  single  project  teams,  whose  parallel 

partners  failed  to  submit  time  forms  will  be  analyzed.   These  four 

projects  involved  the  following  general  problems: 

I.   The  preliminary  design  of  a  spec i a  I -purpose  manned  space- 
craft for  cislunar  missions. 


2.  The  preliminary  design  of  an  earth  orbiting  space 
I  aboratory . 

3.  The  development  of  a  mathematical  model  for  an  interplane- 
tary transportation  system. 

4.  The  development  of  a  container  for  cryogenic  fluids. 


Un  i t  of  Ana  lysi  s 

Man-hours  devoted  to  an  information  channel  is  the  unit  of  ana- 
lysis employed.   (cf.  Menzel,  I960).   Since  engineering  man-hours  ex- 
pended is  quite  closely  correlated  with  cost  on  such  projects  as  these, 
the  choice  of  unit  provides  a  measure  not  only  of  the  relative  extent 
to  which  channels  are  used,  but  also  of  a  major  portion  of  the  rela- 
tive expense  involved  in  such  use. 

Data  Ana  lysi  s 

In  comparing  time  allocated  to  information  channels  across  pro- 
jects, the  variables  in  question  are  represented  as  percentages  of 
total  time.   This  serves  several  purposes.   First  of  all,  the  pro- 
jects are  of  varying  size  and  using  actual  hours  to  represent  the 
variables  would  give  undue  weight  to  the  larger  projects.  Secondly, 

percentage  representation  overcomes  the  problem  of  incomplete  data  in 

2 
any  section  of  a  project.   Finally,  the  total  number  of  engineers  re- 
porting varies  from  team  to  team  within  projects;  and  percentage  re- 
presentation allows  intraproject  comparison.   Percentages  for  any  given 


2 
Due  to  occasional  fai  lure  of  engineers  to  maintain  their  time 
a  I locat  ion  forms. 


time  period  of  a  project  are  the  ratio  of  the  total  time  spent  by 

all  individuals  on  the  variable  in  question  to  the  total  time  reported 

by  a  I  I  i  nd  i V  idua I s. 

RESULTS 

Relations  Between  Performance  and  Time  Allocation 

Higher  and  lower  rated  project  teams  are  determined  on  the  basis 
of  the  government  technical  monitors'  performance  ratings.   The  per- 
cent of  time  spent  in  total  information  gathering,  and  in  literature 
search,  outside  consultation,  staff  consultation,  and  analytic  design 
are  averaged  for  the  higher  and  lower  rated  teams.   The  results  in 
Table  I  show  that  on  the  average  the  higher  rated  teams  spend  more 
time  in  information  gathering.   Literature  search  and  staff  consul- 
tation show  the  greatest  difference. 

Hypotheses  4  and  5,  which  are  based  upon  the  results  of  Allen's 
(1964)  study  of  Ri,D  proposal  competitions  and  on  Shilling  and  Bernard's 
(1964)  findings  that  the  use  of  outside  consultants  by  industrial 
bio-scientists  was  negatively  related  to  eight  "measures  of  laboratory 
productivity  and  efficiency",  are  only  weakly  supported  by  the  evi- 
dence.  Higher  performing  teams  spend  more  time  in  discussions  with  the 
technical  staff  and  lower  performers  spend  more  time  with  external  sources. 
The  difference  between  high  and  low  performers,  however,  is  in  neither 
case  statistically  significant. 


Table   I 

Time  Allocation  by  Function 
(Five  Parallel  Sets  of  R&D  Projects  Plus  Four  Single  Projects) 


F  i  ve  F  i  ve  Tota I  Tota I 
Higher  Lower  for  five  for  14 
Rated  Rated  parallel  projects 
Projects  Projects  sets  

All  information  gathering    19,31  14,74  17,02  15.43 

Literature  search            8,46  5.67  7.06  6.70 

Consultation  with  external    4.60  4.85  4.72  4.22 
information  sources 

Consultation  with            5.77  2.23  4,00  4.51 
technical  staff 

ana  lyt  i  c  des  i  gn 

other 


3.70 

72.94 

73.32 

76.  10 

7.47 

14.31 

10.89 

8.47 

Necessity  of  Analysis  Over  Time 

Figure  I  shows  that  the  time  spent  in  gathering  information  varies 
considerably  over  a  project's  life  and  the  components  that  make  up  the 
total  time  spent  gathering  information  vary  in  a  somewhat  different 
manner,  although  the  interaction  term  is  not  statistically  significant. 
Information  gathering  is  greatest  in  the  initial  time  period.   Litera- 
ture search  and  outside  consulting  make  up  the  large  component  in  this 
state,  corraborat i ng  the  general  belief  that  initially  on  a  new  project 
engineers  conduct  an  extensive  literature  search  and  seek  out  possible 
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Figure  I.   Percent  of  total  time  which  is  spent  in  gathering 
information  as  a  function  of  project  life  span 

(Averaged  for  five  matched  pairs  of  R  8»  D  projects 
plus  4  single  projects) 
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suppliers  for  ideas  as  well  as  components. 

While  literature  search  is  the  primary  information  gathering  tool 
in  the  initial  stages,  it  quicl<ly  decreases  as  the  projects  progress 
and  is  replaced  in  importance  by  person-to-person  communications.   Out- 
side consulting  is  relatively  high  initially,  but  generally  speaking 
is  an  important  factor  in  ail  phases  of  the  projects.   Staff  consulta- 
tion is  greater  at  the  beginning  and  end  of  the  project.   This  seems 
to  indicate  that  the  internal  technical  staff  is  used  least  at  the 
midpoint  of  a  project  and  largely  utilized  during  initial  and  final 
stages. 

Performance  Relations  Over  Time 

Total  Information  Gathering.   When  the  teams  are  again  divided 
on  the  basis  of  high  and  low  performance,  it  is  apparent  (Figure  2) 
that  poor  performers  spend  more  time  in  the  initial  stage  of  a  pro- 
ject than  higher  rated  teams,  but  less  time  in  the  final  stage.   The 
results  are  exactly  counter  to  hypothesis  3  (page  2).   It  was  pre- 
dicted that  those  teams  which  conduct  an  extensive  information  search 
at  the  outset  of  a  project  would  perform  better  as  a  result.   The  re- 
sults, however,  lead  to  the  conclusion  that  the  teams  performing  the  more 
extensive  information  search  were  less  well  prepared  at  the  outset,  per- 
haps through  lack  of  experience  or  failure  to  bring  themselves  up  to 
an  adequate  state  of  knowledge  prior  to  contract  initiation,  and  that 
the  more  intense  information  gathering  in  the  initial  period  fails  to 
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compensate  for  this  deficiency.   As  the  project  progresses,  the  higher 
rated  teams  spend  an  almost  constant  level  of  effort  in  gathering  in- 
formation, and  end  with  a  surge  in  the  final  period.   The  lower  rated 
teams,  on  the  other  hand,  taper  off  markedly  after  the  initial  period 
and  perform  hardly  any  information  gathering  in  the  final  period.   The 
difference  in  the  final  period  may  well  reflect  the  effects  of  pre- 
mature closure  on  the  part  of  the  poorer  performers.   Since  all  of  the 
projects  were  studies  rather  than  hardware  developments,  it  may  well 
be  beneficial,  in  such  cases,  to  remain  receptive  to  new  Ideas  right 
up  to  the  time  when  the  study  report  is  delivered  to  the  customer. 
The  better  performing  teams  apparently  behaved  in  this  manner,  actually 
devoting  an  increased  amount  of  time  to  information  gathering  In  the 
f I na  I  period , 

Outside  Sources.   Higher  rated  teams  are  far  more  consistent  over 
time  than  lower  rated  teams  in  their  use  of  information  sources  out- 
side the  laboratory.   The  higher  rated  teams  seem  to  stay  closer  to 
the  ideal  of  bringing  outsiders  "on  board"  early  and  maintaining  close 
contact  throughout  the  project's  life,  not  just  when  difficulties  arise. 
In  this  way  both  parties  are  kept  sufficiently  aware  of  progress  made 
by  the  other,  and  such  difficulties  as  interface  incompatibility  are 
reduced.   The  lower  rated  teams,  on  the  other  hand,  do  not  maintain 
close  liaison  with  the  external  sources,  and  periodic  spurts  of  heavy 
communication  fail  to  compensate  for  the  lack  of  a  continuous  relationship. 
(Figure  3). 
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Figure  3  Comparison  of  percent  time  spent  by  higher  and  lower 
rated  project  teams  in  consulting  with  sources  of 
information  outside  the  laboratory  (Averaged  for  five 
matched  pairs  of  R  8i  D  projects) 
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Technical  Staff.   Although  the  time  spent  with  this  channel  is 
the  smallest  reported  (2.23^  for  losing  teams  and  b .11%   for  higher 
rated  teams),  it  is  a  very  important  source  of  technical  information. 
Staff  specialists  are  often  called  upon  to  solve  critical  problems 
beyond  the  capability  of  team  members.   Again  the  time  pattern  fol- 
lows that  established  for  outside  sources  —  consistency  over  time 
by  the  higher  rated  teams  and  fluctuation  by  the  lower  rated  teams. 

The  higher  rated  teams  utilized  staff  assistance  from  the 
beginning  of  the  project  to  the  end  and  maintained  contact  with  this 
source  of  information  throughout  the  duration  of  the  project  (Figure 
4).   The  lower  rated  teams  call  for  staff  assistance  at  the  beginning 
of  the  project,  then  drop  to  very  low  utilization  of  staff  in  the 
middle  (third  and  fourth  sextile,  .43).   Toward  the  end,  perhaps 
recognizing  difficulties  the  staff  is  again  called  in.   A  practice 
often  employed  by  industry  is  to  utilize  staff  assistance  as  an 
information  source  only  when  difficulties  arise.   The  major  draw- 
back of  this  procedure  is  that  it  is  often  too  late  for  the  staff 
to  be  of  appreciable  assistance.   This  research  confirms  the  fact 
that  higher  rated  teams  utilize  the  staff  quite  evenly  throughout  the 
duration  of  the  project  which  keeps  this  valuable  information  channel 
open  during  both  good  and  bad  times,  possibly  forestalling  some  dif- 
ficulties before  they  have  time  to  become  established. 

Literature  Search.   Although  the  differences  between  higher  and 
lower  rated  teams  are  not  significant,  the  general  pattern  is  similar 
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Figure  4  Comparison  of  percent  time  spent  by  higher  end 
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to  that  for  the  other  information  channels.   Higher  rated  teams  re- 
main fairly  steady  and  lower  rated  teams  fluctuate.  The  lower  rated 
teams,  however,  are  not  quite  as  erratic  as  in  other  information  acti- 
vities, showing  instead  a  rather  steady  decline  as  the  project  pro- 
gresses (Figure  5). 

Uncertain  Subsystems 

When  there  is  a  great  uncertainty  about  the  outcome  of  the  solu- 
tion approaches  under  consideration,  engineers  will  either  seek  out 
new  approaches  or  try  to  reduce  the  uncertainty  in  the  old  ones  through 
attainment  of  greater  knowledge.   In  either  case  information  is  gathered 
to  aid  their  decision.   Therefore,  it  was  hypothesized  that  subsystems 
characterized  by  high  uncertainty  would  require  more  information  gather- 
ing than  those  in  which  uncertainty  is  low. 

To  test  this,  it  is  first  necessary  to  define  and  measure  uncer- 
tainty.  At  least  two  estimates  of  uncertainty  are  available  from 

the  data.   From  the  Solution  Development  Record,  the  average  weekly 

3 
value  for  Ap  can  be  determined  for  each  subsystem.   A  large  value 

of   Ap  should  indicate  high  uncertainty  but  in  some  cases  several 
alternatives  are  carried  at  a  low  level  throughout  most  of  the  pro- 
ject with  none  achieving  clear  dominance;  in  such  cases  Ap  can  be 


3 
Engineers  provide  a  weekly  estimate,  on  the  Solution  Development 
Records,  of  the  subjective  probability  that  each  alternative  considered 
as  a  possible  solution  will  in  fact  be  finally  chosen  as  the  solution. 
The  measure   A  p  is  the  total  change  in  probability  level  for  all 
alternatives  in  a  given  subproblem  (Allen,  1966). 
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Figure  5  Comparison  of  percent  time  spent  by  higher  and 
lower  rated  project  teams  in  literature  search. 
(Averaged  for  five  matched  pairs  of  R  &  D  projects) 
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very  small  and  yet  the  uncertainty  is  obviously  high.  A  more  reliable 
and  sensitive  measure  appears  to  be  the  number  of  weeks  the  winning 
alternative  maintains  a  probability  less  than  0.7  on  the  Solution 
Development  Record.   If  the  winning  alternative  is  reported  at  a  pro- 
bability of  0.7  or  higher  throughout  most  of  the  project,  the  sub- 
system is  one  with  a  relatively  high  degree  of  certainty.   Conversely, 
if  the  probability  of  acceptance  exceeds  0.7  for  only  a  small  portion 
of  the  project,  this  would  indicate  uncertainty  relative  to  the  design 
of  the  subsystem. 

Uncertain  subsystems  are  thus  defined  in  the  following  two  ways: 

(1)  Those  whose  final  solution  maintained  a  0.7  or  higher  probability 
for  a  number  of  weeks  which  was  less  than  the  average  for  the  project. 

(2)  Those  with  a  weekly  Ap  greater  than  the  average  for  the  project. 

The  three  operations  research  projects  and  one  of  the  other  pro- 
jects, which  was  oriented  more  toward  basic  research  than  development, 
are  omitted  from  this  analysis  because  in  these  instances  subsystems 
are  not  clearly  defined,  and  relative  uncertainty  is  therefore  impos- 
sible to  determine.   In  each  of  the  other  two  projects,  it  is  possible 
to  identify  thirty-one  individual  subsystem  designs.   There  is  some 
overlap  among  individuals  in  their  responsibility  for  subsystems;  in 
these  cases  assignment  is  made  equally  to  each.   This  is  a  conservative 
procedure  which  results  in  a  general  weakening  of  correlations  for  the 
subsystems  affected.   In  the  case  of  one  parallel  program  the  overlap 
prevented  identification  of  responsibilities  for  subsystems  so  these 
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data  are  not  included.   For  the  basic  research  project  where  subsys- 
tem identification  was  not  clear,  the  laboratory  that  snowed  greater 
uncertainty  ower    its  approaches  spent  41.10^  of  its  time  in  informa- 
tion gathering.   The  laboratory  that  was  more  certain  over  its  ap- 
proaches spent  20^  of  its  time  in  information  gathering. 

With  these  limitations  noted,  the  percent  of  information  gather- 
ing devoted  to  each  subsystem  is  compared  with  the  project  average. 
Figure  6  shows  the  results  of  chi-squared  tests  of  the  hypothesis 
for  each  of  the  definitions  of  an  uncertain  subsystem. 

The  results  indicate  rather  strongly  that  engineers  allocate 
more  time  to  information  gathering  when  they  are  working  on  problems 
of  higher  uncertainty.   This  being  the  case,  subsystem  uncertainty 
measures  should  be  determined  at  project  initiation,  so  that  engineers 
responsible  for  subsystems  with  the  higher  degrees  of  uncertainty 
may  be  provided  with  greater  ease  of  access  to  information  channels. 
Total  information  gathering  should  De  planned,  budgeted  and  allocated 
on  the  basis  of  subsystem  uncertainty  measures.   Literature  access 
in  the  form  of  information  retrieval  systems  and  library  assistance 
should  be  readily  available  to  these  subsystem  engineers.   Provision 
should  be  made  for  assigning  consultants  from  the  technical  staff  to 
such  subsystems  at  an  early  enough  stage  to  avoid  the  apparently  poor 
and  intermittent  contacts  observed  on  the  poorer  performing  projects 
(f  igure  4) . 
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Certainty  based  upon  number  of  weeks  in  which  final  solution 
exceeded  0.7  probability  of  acceptance.* 


uncerta  i  n 
subsystems 


certai  n 
subsystems 


Percent  of     High 
Total  Time 
Spent  in 
I nf ormat  ion 
Gathering       Low 


12 

6 

3 

9 

probabi  I ity  of 
these  results 
occuring  by  chance 
are  less  than 
0.05. 


X  =  5.01 
p<.05 


*N  reduced  by  one  as  a  result  of  discarding  one  subsystem 
which  fell  on  the  median  of  the  uncertainty  measure. 


Certainty  based  upon  weekly  probability  change. 
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are  less  than 
0.10. 


X   =  3.60 
p<.  10 


Figure  6.   Proportion  of  time  devoted  to  information  gathering 
as  a  function  of  subsystem  certainty. 


CONCLUSIONS 

Five  pairs  of  government  funded  parallel  research  and  development 
projects  and  four  single  projects  are  examined  to  determine  how  their 
engineers  and  scientists  allocate  time,  and  the  effect  of  this  allo- 
cation on  the  outcome  of  the  projects.   The  use  of  matched  pairs  of 
projects  allows  the  relative  evaluation  of  outcomes  by  technical  moni- 
tors in  the  customer  agencies. 

Three  relationships  are  investigated:   the  percent  time  spent  in 
three  information  gathering  activities  as  a  function  of  the  phase  of 
the  project;  the  relation  of  time  allocation  patterns  to  performance; 
and  the  correlation  between  degree  of  uncertainty  in  a  subsystem  and 
the  amount  of  information  gathering  devoted  to  that  subsystem. 

The  percent  of  total  time  spent  In  all  phases  of  information 
gathering  varies  significantly  over  the  duration  of  projects.   When 
the  activities  are  measured  over  time,  a  clear  relation  with  perfor- 
mance emerges.   Higher  rated  teams  are  more  consistent  over  the  dura- 
tion of  the  projects  in  all  phases  of  information  gathering,  while 
lower  rated  teams  initially  spend  more  time  gathering  information 
than  they  do  in  the  later  stages  and  fluctuate  more  throughout.   Lit- 
erature search  by  lower  rated  teams  is  regular  in  its  declining  use 
from  the  beginning  to  the  end  of  projects.   Staff  activity  for  lower 
rated  teams  drops  sharply  in  the  middle  of  the  project  suggesting  the 
beneficial  effects  of  more  continuous  contact  with  the  internal  staff. 
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These  results  hold  for  two  methods  of  computing  percent  time  spent  in 
any  activity. 

The  time  of  RiD  engineers  is  a  most  valuable  commodity.  Much  of 
this  time  is  spent  in  seeking  and  gathering  needed  technical  information, 
Certain  procedures  should  be  designed  to  increase  the  efficiency  of 
this  search  in  all  instances,  but  where  a  choice  must  be  made,  steps 
should  be  first  taken  in  areas  where  the  problem  is  most  critical.   The 
present  research  indicates  that  one  mechanism  for  determining  such  cri- 
tical ity  is  an  estimate  of  the  relative  uncertainty  of  solution  of  each 
subproblem.   Since  more  uncertain  subproblems  are  in  greater  need  of 
technical  information,  the  engineers  assigned  to  such  subproblems 
should  be  provided  special  assistance  in  gathering  needed  information. 
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